Artificial Intelligence Nanodegree¶
Voice User Interfaces¶
Project: Speech Recognition with Neural Networks¶
In this notebook, some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this project. You will not need to modify the included code beyond what is requested. Sections that begin with '(IMPLEMENTATION)' in the header indicate that the following blocks of code will require additional functionality which you must provide. Please be sure to read the instructions carefully!
Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the Jupyter Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.
In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question X' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.
The rubric contains optional "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. If you decide to pursue the "Stand Out Suggestions", you should include the code in this Jupyter notebook.
Introduction¶
In this notebook, you will build a deep neural network that functions as part of an end-to-end automatic speech recognition (ASR) pipeline! Your completed pipeline will accept raw audio as input and return a predicted transcription of the spoken language. The full pipeline is summarized in the figure below.
- STEP 1 is a pre-processing step that converts raw audio to one of two feature representations that are commonly used for ASR.
- STEP 2 is an acoustic model which accepts audio features as input and returns a probability distribution over all potential transcriptions. After learning about the basic types of neural networks that are often used for acoustic modeling, you will engage in your own investigations, to design your own acoustic model!
- STEP 3 in the pipeline takes the output from the acoustic model and returns a predicted transcription.
Feel free to use the links below to navigate the notebook:
- The Data
- STEP 1: Acoustic Features for Speech Recognition
- STEP 2: Deep Neural Networks for Acoustic Modeling
- Model 0: RNN
- Model 1: RNN + TimeDistributed Dense
- Model 2: CNN + RNN + TimeDistributed Dense
- Model 3: Deeper RNN + TimeDistributed Dense
- Model 4: Bidirectional RNN + TimeDistributed Dense
- Models 5+
- Compare the Models
- Final Model
- STEP 3: Obtain Predictions
from importlib import reload
# watch for any changes in the sample_models module, and reload it automatically
%load_ext autoreload
%autoreload 2
# import NN architectures for speech recognition
import models as M
# import function for training acoustic model
import train_utils as T
import utils as U
import data_generator as DG
print()
# U.config_GPU(gpu_memory_fraction=0.4)
U.config_GPU(allow_growth = True)
from keras.optimizers import SGD
Audio Data Generator¶
def reload_all():
reload(U)
reload(DG)
reload(M)
reload(T)
audio_gen_mfcc = None
audio_gen_spec = None
audio_gen_raw = None
audio_gen_mfcc_demo = None
audio_gen_spec_demo = None
audio_gen_raw_demo = None
def init_gen(spectrogram=True, raw=False, shuffle=True):
print("Initializing Generator for ",
"Raw" if raw else
("MFCC" if not spectrogram else "Spectrogram"), "No Shuffle" if not shuffle else "")
audio_gen = DG.AudioGeneratorCached(spectrogram=spectrogram, raw=raw, minibatch_size=32, shuffle_data=shuffle)
audio_gen.load_train_data('train_corpus.json')
audio_gen.load_validation_data('valid_corpus.json')
return audio_gen
def init_gen_var(spectrogram=True, raw=False, shuffle=True):
global audio_gen_mfcc
global audio_gen_spec
global audio_gen_raw
global audio_gen_mfcc_demo
global audio_gen_spec_demo
global audio_gen_raw_demo
# print("init_gen_var", spectrogram, raw, shuffle)
if shuffle:
if raw:
if not audio_gen_raw:
audio_gen_raw = init_gen(raw=True, shuffle=True)
elif spectrogram:
if not audio_gen_spec:
audio_gen_spec = init_gen(spectrogram=True, shuffle=True)
else:
if not audio_gen_mfcc:
audio_gen_mfcc = init_gen(spectrogram=False, shuffle=True)
else:
if raw:
if not audio_gen_raw_demo:
audio_gen_raw_demo = init_gen(raw=True, shuffle=False)
elif spectrogram:
if not audio_gen_spec_demo:
audio_gen_spec_demo = init_gen(spectrogram=True, shuffle=False)
else:
if not audio_gen_mfcc_demo:
audio_gen_mfcc_demo = init_gen(spectrogram=False, shuffle=False)
def get_gen(spectrogram=True, raw=False, shuffle=True):
init_gen_var(spectrogram, raw, shuffle)
if shuffle:
return audio_gen_raw if raw else (audio_gen_spec if spectrogram else audio_gen_mfcc)
else:
return audio_gen_raw_demo if raw else (audio_gen_spec_demo if spectrogram else audio_gen_mfcc_demo)
model_names_to_compare = []
def train_model(model_builder, *args, spectrogram=True, raw=False, mfcc_concat=False, **kwargs):
reload(M)
reload(T)
global audio_gen_mfcc
global audio_gen_spec
global audio_gen_raw
init_gen_var(spectrogram)
name = T.train_model(audio_gen_raw if raw else (audio_gen_spec if spectrogram else audio_gen_mfcc), model_builder, *args, **kwargs)
model_names_to_compare.append(name)
def plot_comparison(model_names=None, *args, **kwargs):
reload(T)
if model_names is None:
names = model_names_to_compare
print_model_names(names)
else:
names = model_names
pickles = []
for name in names:
pickles.append("results/" + name + ".pickle")
T.plot_comparison(pickles=pickles, *args, **kwargs)
def print_model_names(model_names=None):
print_list_of_strings(model_names if model_names is not None else model_names_to_compare)
def print_list_of_strings(list_of_strings):
print("[", end="")
first = True
for name in list_of_strings:
if not first:
print(",")
print(" ", end="")
else:
first = False
print("\'"+name+"\'", end="")
print("]")
reload(DG)
DG.test_gen()
The Data¶
We begin by investigating the dataset that will be used to train and evaluate your pipeline. LibriSpeech is a large corpus of English-read speech, designed for training and evaluating models for ASR. The dataset contains 1000 hours of speech derived from audiobooks. We will work with a small subset in this project, since larger-scale data would take a long while to train. However, after completing this project, if you are interested in exploring further, you are encouraged to work with more of the data that is provided online.
In the code cells below, you will use the vis_train_features module to visualize a training example. The supplied argument index=0 tells the module to extract the first example in the training set. (You are welcome to change index=0 to point to a different training example, if you like, but please DO NOT amend any other code in the cell.) The returned variables are:
vis_text- transcribed text (label) for the training example.vis_raw_audio- raw audio waveform for the training example.vis_mfcc_feature- mel-frequency cepstral coefficients (MFCCs) for the training example.vis_spectrogram_feature- spectrogram for the training example.vis_audio_path- the file path to the training example.
# extract label and audio features for a single training example
vis_text, vis_raw_audio, vis_mfcc_feature, vis_spectrogram_feature, vis_audio_path = DG.vis_train_features(mfcc_classes=26)
The following code cell visualizes the audio waveform for your chosen example, along with the corresponding transcript. You also have the option to play the audio in the notebook!
from IPython.display import Markdown, display
from IPython.display import Audio
%matplotlib inline
# plot audio signal
DG.plot_raw_audio(vis_raw_audio)
# print length of audio signal
display(Markdown('**Shape of Audio Signal** : ' + str(vis_raw_audio.shape)));
# print transcript corresponding to audio clip
display(Markdown('**Transcript** : ' + str(vis_text)));
# play the audio file
Audio(vis_audio_path)
STEP 1: Acoustic Features for Speech Recognition¶
For this project, you won't use the raw audio waveform as input to your model. Instead, we provide code that first performs a pre-processing step to convert the raw audio to a feature representation that has historically proven successful for ASR models. Your acoustic model will accept the feature representation as input.
In this project, you will explore two possible feature representations. After completing the project, if you'd like to read more about deep learning architectures that can accept raw audio input, you are encouraged to explore this research paper.
Spectrograms¶
The first option for an audio feature representation is the spectrogram. In order to complete this project, you will not need to dig deeply into the details of how a spectrogram is calculated; but, if you are curious, the code for calculating the spectrogram was borrowed from this repository. The implementation appears in the utils.py file in your repository.
The code that we give you returns the spectrogram as a 2D tensor, where the first (vertical) dimension indexes time, and the second (horizontal) dimension indexes frequency. To speed the convergence of your algorithm, we have also normalized the spectrogram. (You can see this quickly in the visualization below by noting that the mean value hovers around zero, and most entries in the tensor assume values close to zero.)
# plot normalized spectrogram
DG.plot_spectrogram_feature(vis_spectrogram_feature)
# print shape of spectrogram
display(Markdown('**Shape of Spectrogram** : ' + str(vis_spectrogram_feature.shape)));
Mel-Frequency Cepstral Coefficients (MFCCs)¶
The second option for an audio feature representation is MFCCs. You do not need to dig deeply into the details of how MFCCs are calculated, but if you would like more information, you are welcome to peruse the documentation of the python_speech_features Python package. Just as with the spectrogram features, the MFCCs are normalized in the supplied code.
The main idea behind MFCC features is the same as spectrogram features: at each time window, the MFCC feature yields a feature vector that characterizes the sound within the window. Note that the MFCC feature is much lower-dimensional than the spectrogram feature, which could help an acoustic model to avoid overfitting to the training dataset.
# plot normalized MFCC
DG.plot_mfcc_feature(vis_mfcc_feature)
# print shape of MFCC
display(Markdown('**Shape of MFCC** : ' + str(vis_mfcc_feature.shape)));
When you construct your pipeline, you will be able to choose to use either spectrogram or MFCC features. If you would like to see different implementations that make use of MFCCs and/or spectrograms, please check out the links below:
- This repository uses spectrograms.
- This repository uses MFCCs.
- This repository also uses MFCCs.
- This repository experiments with raw audio, spectrograms, and MFCCs as features.
STEP 2: Deep Neural Networks for Acoustic Modeling¶
In this section, you will experiment with various neural network architectures for acoustic modeling.
You will begin by training five relatively simple architectures. Model 0 is provided for you. You will write code to implement Models 1, 2, 3, and 4. If you would like to experiment further, you are welcome to create and train more models under the Models 5+ heading.
All models will be specified in the sample_models.py file. After importing the sample_models module, you will train your architectures in the notebook.
After experimenting with the five simple architectures, you will have the opportunity to compare their performance. Based on your findings, you will construct a deeper architecture that is designed to outperform all of the shallow models.
For your convenience, we have designed the notebook so that each model can be specified and trained on separate occasions. That is, say you decide to take a break from the notebook after training Model 1. Then, you need not re-execute all prior code cells in the notebook before training Model 2. You need only re-execute the code cell below, that is marked with RUN THIS CODE CELL IF YOU ARE RESUMING THE NOTEBOOK AFTER A BREAK, before transitioning to the code cells corresponding to Model 2.
Model 0: RNN¶
Given their effectiveness in modeling sequential data, the first acoustic model you will use is an RNN. As shown in the figure below, the RNN we supply to you will take the time sequence of audio features as input.
At each time step, the speaker pronounces one of 28 possible characters, including each of the 26 letters in the English alphabet, along with a space character (" "), and an apostrophe (').
The output of the RNN at each time step is a vector of probabilities with 29 entries, where the $i$-th entry encodes the probability that the $i$-th character is spoken in the time sequence. (The extra 29th character is an empty "character" used to pad training examples within batches containing uneven lengths.) If you would like to peek under the hood at how characters are mapped to indices in the probability vector, look at the char_map.py file in the repository. The figure below shows an equivalent, rolled depiction of the RNN that shows the output layer in greater detail.
The model has already been specified for you in Keras. To import it, you need only run the code cell below.
As explored in the lesson, you will train the acoustic model with the CTC loss criterion. Custom loss functions take a bit of hacking in Keras, and so we have implemented the CTC loss function for you, so that you can focus on trying out as many deep learning architectures as possible :). If you'd like to peek at the implementation details, look at the add_ctc_loss function within the train_utils.py file in the repository.
To train your architecture, you will use the train_model function within the train_utils module; it has already been imported in one of the above code cells. The train_model function takes three required arguments:
input_to_softmax- a Keras model instance.pickle_path- the name of the pickle file where the loss history will be saved.save_model_path- the name of the HDF5 file where the model will be saved.
If we have already supplied values for input_to_softmax, pickle_path, and save_model_path, please DO NOT modify these values.
There are several optional arguments that allow you to have more control over the training process. You are welcome to, but not required to, supply your own values for these arguments.
minibatch_size- the size of the minibatches that are generated while training the model (default:20).spectrogram- Boolean value dictating whether spectrogram (True) or MFCC (False) features are used for training (default:True).mfcc_dim- the size of the feature dimension to use when generating MFCC features (default:13).optimizer- the Keras optimizer used to train the model (default:SGD(lr=0.02, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5)).epochs- the number of epochs to use to train the model (default:20). If you choose to modify this parameter, make sure that it is at least 20.verbose- controls the verbosity of the training output in themodel.fit_generatormethod (default:1).sort_by_duration- Boolean value dictating whether the training and validation sets are sorted by (increasing) duration before the start of the first epoch (default:False).
The train_model function defaults to using spectrogram features; if you choose to use these features, note that the acoustic model in simple_rnn_model should have input_dim=161. Otherwise, if you choose to use MFCC features, the acoustic model should have input_dim=13.
We have chosen to use GRU units in the supplied RNN. If you would like to experiment with LSTM or SimpleRNN cells, feel free to do so here. If you change the GRU units to SimpleRNN cells in simple_rnn_model, you may notice that the loss quickly becomes undefined (nan) - you are strongly encouraged to check this for yourself! This is due to the exploding gradients problem. We have already implemented gradient clipping in your optimizer to help you avoid this issue.
IMPORTANT NOTE: If you notice that your gradient has exploded in any of the models below, feel free to explore more with gradient clipping (the clipnorm argument in your optimizer) or swap out any SimpleRNN cells for LSTM or GRU cells. You can also try restarting the kernel to restart the training process.
train_model(M.RNNModel(bd_merge=None, rnn_type = M.RNNType.SimpleRNN, time_distributed_dense=False),
spectrogram=False,
epochs=20, loss_limit=None)
train_model(M.RNNModel(bd_merge=None, rnn_type = M.RNNType.SimpleRNN, time_distributed_dense=False),
epochs=20, loss_limit=None)
train_model(M.RNNModel(bd_merge=None, time_distributed_dense=False),
epochs=20, loss_limit=None)
Observations:¶
- Clearly, a SimpleRNN or LSTM models without Time-Distributed Dense layers are inadequate
(IMPLEMENTATION) Model 1: RNN + TimeDistributed Dense¶
Read about the TimeDistributed wrapper and the BatchNormalization layer in the Keras documentation. For your next architecture, you will add batch normalization to the recurrent layer to reduce training times. The TimeDistributed layer will be used to find more complex patterns in the dataset. The unrolled snapshot of the architecture is depicted below.
The next figure shows an equivalent, rolled depiction of the RNN that shows the (TimeDistrbuted) dense and output layers in greater detail.
Use your research to complete the rnn_model function within the sample_models.py file. The function should specify an architecture that satisfies the following requirements:
- The first layer of the neural network should be an RNN (
SimpleRNN,LSTM, orGRU) that takes the time sequence of audio features as input. We have addedGRUunits for you, but feel free to changeGRUtoSimpleRNNorLSTM, if you like! - Whereas the architecture in
simple_rnn_modeltreated the RNN output as the final layer of the model, you will use the output of your RNN as a hidden layer. UseTimeDistributedto apply aDenselayer to each of the time steps in the RNN output. Ensure that eachDenselayer hasoutput_dimunits.
train_model(M.RNNModel(bd_merge=None, rnn_type=RNNType.LSTM))
Observations:¶
- Addition of Time-Distributed Dense drastically improves performance
(IMPLEMENTATION) Model 2: CNN + RNN + TimeDistributed Dense¶
The architecture in cnn_rnn_model adds an additional level of complexity, by introducing a 1D convolution layer.
This layer incorporates many arguments that can be (optionally) tuned when calling the cnn_rnn_model module. We provide sample starting parameters, which you might find useful if you choose to use spectrogram audio features.
If you instead want to use MFCC features, these arguments will have to be tuned. Note that the current architecture only supports values of 'same' or 'valid' for the conv_border_mode argument.
When tuning the parameters, be careful not to choose settings that make the convolutional layer overly small. If the temporal length of the CNN layer is shorter than the length of the transcribed text label, your code will throw an error.
Before running the code cell below, you must modify the cnn_rnn_model function in sample_models.py. Please add batch normalization to the recurrent layer, and provide the same TimeDistributed layer as before.
train_model(model_builder=M.RNNModel(cnn_config=M.CNNConfig(), bd_merge=None, rnn_type=RNNType.LSTM))
plot_comparison(model_names=[
'Spec CuDNNLSTM(200 x1) (relu) BN DO(0.2) TD(D)',
'Spec CNN(200(11,2) DO(0.2) BN CuDNNLSTM(200 x1) (relu) BN DO(0.2) TD(D)'
], max_loss=160)
Observations:¶
- Addition of CNN Layer before the RNN part improves performance considerably
(IMPLEMENTATION) Model 3: Deeper RNN + TimeDistributed Dense¶
Review the code in rnn_model, which makes use of a single recurrent layer. Now, specify an architecture in deep_rnn_model that utilizes a variable number recur_layers of recurrent layers. The figure below shows the architecture that should be returned if recur_layers=2. In the figure, the output sequence of the first recurrent layer is used as input for the next recurrent layer.
Feel free to change the supplied values of units to whatever you think performs best. You can change the value of recur_layers, as long as your final value is greater than 1. (As a quick check that you have implemented the additional functionality in deep_rnn_model correctly, make sure that the architecture that you specify here is identical to rnn_model if recur_layers=1.)
train_model(M.RNNModel(bd_merge=None, rnn_type=RNNType.LSTM, rnn_layers=2))
train_model(M.RNNModel(bd_merge=None, rnn_type=RNNType.LSTM, rnn_layers=3))
plot_comparison(model_names=[
'Spec CuDNNLSTM(200 x1) (relu) BN DO(0.2) TD(D)',
'Spec CuDNNLSTM(200 x2) DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)',
'Spec CuDNNLSTM(200 x3 DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)'
], max_loss=160)
Observations:¶
- Addition of more RNN layers (without CNN layer) improves performance
(IMPLEMENTATION) Model 4: Bidirectional RNN + TimeDistributed Dense¶
Read about the Bidirectional wrapper in the Keras documentation. For your next architecture, you will specify an architecture that uses a single bidirectional RNN layer, before a (TimeDistributed) dense layer. The added value of a bidirectional RNN is described well in this paper.
One shortcoming of conventional RNNs is that they are only able to make use of previous context. In speech recognition, where whole utterances are transcribed at once, there is no reason not to exploit future context as well. Bidirectional RNNs (BRNNs) do this by processing the data in both directions with two separate hidden layers which are then fed forwards to the same output layer.
Before running the code cell below, you must complete the bidirectional_rnn_model function in sample_models.py. Feel free to use SimpleRNN, LSTM, or GRU units. When specifying the Bidirectional wrapper, use merge_mode='concat'.
train_model(M.RNNModel(bd_merge=M.BidirectionalMerge.concat, rnn_layers=2))
train_model(M.RNNModel(bd_merge=M.BidirectionalMerge.sum, rnn_layers=2))
train_model(M.RNNModel(bd_merge=M.BidirectionalMerge.ave, rnn_layers=2))
train_model(M.RNNModel(bd_merge=M.BidirectionalMerge.mul, rnn_layers=2))
plot_comparison(model_names=[
'Spec CuDNNLSTM(200 x2) DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)',
'Spec BD(sum) CuDNNLSTM(200 x2 DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)',
'Spec BD(ave) CuDNNLSTM(200 x2 DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)',
'Spec BD(concat) CuDNNLSTM(200 x2 DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)',
'Spec BD(mul) CuDNNLSTM(200 x2 DO(0.2)(:-1)) (relu) BN DO(0.2) TD(D)'
], min_loss=120, max_loss=160)
Observations:¶
- All bidirectional models perform better than the unidirectional one
- Among the merge modes, multiplication probably shows the worst performance with the remaining three showing roughly similar performance. The concat mode might reach its best slightly sooner.
(OPTIONAL IMPLEMENTATION) Models 5+¶
If you would like to try out more architectures than the ones above, please use the code cell below. Please continue to follow the same convention for saving the models; for the $i$-th sample model, please save the loss at model_i.pickle and saving the trained model at model_i.h5.
Comparing Dropout, Batch Normalization and Activation Configuration of the CNN layer¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_do_bn_order=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_bn=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False, cnn_bn=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False,
cnn_do_bn_order=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False,
cnn_do_bn_order=False, cnn_dropout_rate=0),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=30)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu BN) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) BN relu) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2)) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) DO(0.2) relu) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu BN DO(0.2)) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) DO(0.2) relu BN) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) BN relu DO(0.2)) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=150, min_loss=100, max_epoch=30)
Observations:¶
- Performance of models without dropout layer in CNN part drops considerably, even though the batch normalizaiton is present. So the batch normalization layer does not make dropout unnecessary.
- If batch normalization is omitted, with dropout present, performance also degrades but to lesser extent. So the dropout seems to be even more essential than batch normalization
- Once both dropout and batch normalization are present the order or ReLU layer, batch normalization and dropout does NOT matter. This is surprising. The batch normalization paper makes a great deal of the batch norm layer being followed by an activation function. What we see here is that the batch norm does help, but is can be put after the activation function and even after a dropout layer and that does not affect the model performance
Comparing effect of order of Dropout, Batch Normalization and Activation for two CNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_layers=2),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_do_bn_order=False, cnn_layers=2),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False, cnn_layers=2),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_activation_before_bn_do=False,
cnn_do_bn_order=False, cnn_layers=2),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(max_loss=200, min_loss=100)
Observations:¶
- With two CNN layers, the order of batch norm, ReLU and dropout again does not matter much. The recommended configuration BN-ReLU-DO performs, if anything, does worse than the others
Checking the effect of adding CNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=11, conv_stride=2, conv_border_mode="valid",
cnn_layers=1,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=4,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=5,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) DO(0.2) relu BN) BD(concat) CuDNNGRU(200 x1) BN relu DO(0.2) TD(D)',
'Spec CNN(200 (5,1) DO(0.2) relu BN)x2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (5,1) DO(0.2) relu BN)x4 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x5 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x8 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=150, min_loss=90, max_epoch=50)
Observations:¶
- Multiple CNN layers with small kernel size perform better than small number of CNN layers with larger kernels. This is well-known for image-processing CNNs. Stacked small kernels have the same effective span as few layers with larger kernels yet the former are more flexible than the latter due to more linear transformations and non-linear activations.
Determining the optimal number of CNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=10,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=16,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=150)
plot_comparison(model_names=\
['Spec CNN(200 (5,1) DO(0.2) relu BN)x4 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x5 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x8 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x10 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x16 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=120, min_loss=90, max_epoch=150)
Observations:¶
- In models with one RNN layer and time distributed dense, 10 to 12 preceding CNN layers appear to show best performance.
Testing the order of Dropout, Batch Normalization and Activation for three and eight CNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=3,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=3,
cnn_activation_before_bn_do=False,
cnn_do_bn_order=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=3),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(max_loss=150, min_loss=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8,
cnn_activation_before_bn_do=False,
cnn_do_bn_order=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8,
cnn_activation_before_bn_do=False,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
plot_comparison(max_loss=120, min_loss=75)
Observations:¶
- With three or eight CNN layers, again, the order of batch norm, ReLU and dropout again does not matter much with the recommended configuration BN-ReLU-DO appearing to do worst
Testing the impact of DenseNet architecture on the CNN part of the model¶
The DenseNet architecture promises faster training, narrower layers and better results. In the DenseNet paper it was used to achieve state-of-the-art performance of deep CNN networks. The output of every DenseNet layer is concatenated in the input of every following layer.
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dense=True, filters=76),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
from keras.optimizers import SGD
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dense=True, filters=76),
rnn_layers=1),
optimizer=SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5),
epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dense=True, filters=76, cnn_dropout_rate=0.3),
rnn_layers=1),
optimizer=SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5),
epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dense=True, filters=150),
rnn_layers=1),
optimizer=SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5),
epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=18, cnn_dense=True, filters=76),
rnn_layers=1),
optimizer=SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5),
epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dense=True, filters=90),
rnn_layers=0),
optimizer=SGD(lr=0.05, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5),
epochs=30)
plot_comparison(model_names=\
['Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN_DENSE(76 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.2) TD(D)',
'Spec CNN_DENSE(76 (3,1) DO(0.2) relu BN)x18 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.2) TD(D)',
'Spec CNN_DENSE(76 (3,1) DO(0.3) relu BN)x12 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.2) TD(D)'],
max_loss=150, min_loss=90, max_epoch=80)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=2, cnn_dense=True, filters=100,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=5, conv_stride=1, conv_border_mode="same",
cnn_layers=4, cnn_dense=True, filters=80,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=5, cnn_dense=True, filters=80,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8, cnn_dense=True, filters=50,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(
['Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x8 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN_DENSE(80 (5,1) DO(0.2) relu BN)x4 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN_DENSE(80 (3,1) DO(0.2) relu BN)x5 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN_DENSE(50 (3,1) DO(0.2) relu BN)x8 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=150, min_loss=75, max_epoch=50)
Observations:¶
- The blue and yellow lines on the chart above are the models without DenseNet connections in the CNN part
- Dense CNN may train faster for this task but, contrary to expectations, show worse performance than non-dense CNN
Testing the effect of Dilations in CNN¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=2, dilation=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=3, dilation=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=4, dilation=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=5, dilation=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=6, dilation=2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
More than 6 layers with dilation=2 does not work
plot_comparison(model_names=
['Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x2,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x3,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x4,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x5,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x6,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=150, min_loss=90, max_epoch=100)
It was recommended to look at dilations and at the WaveNet paper in particular. In the WaveNet paper dilations in CNN were applied to raw sound data, not spectrograms as attempted here. In the networks above, the first CNN layer has no dilation, the next has dilation 2, next dilation 4 etc.
Observations:¶
- The blue line above is the model without dilations
- Dilations do not help at all if applied to spectrogram data
Trying Inverse Dilations¶
Below the inverse dilations are tested, where the last CNN layer has dilation 1, previous one has dilation 2 etc. :)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=11, conv_stride=1, conv_border_mode="valid",
cnn_layers=2, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=2, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=3, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=4, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=5, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=6, dilation=-2, # cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(min_loss=80, max_loss=120, max_epoch=50)
Observations:¶
- The blue line above is the model without dilations
- Inverse dilations do not help either
- What is amusing is that inverse dilations, however weird they may seem, perform about as "well" as regular dilations
Testing the order of Batch Normalization, ReLU and Dropout with dilutions¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=4, dilation=2, cnn_do_bn_order=False,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=4, dilation=2, cnn_dropout_rate=0.4,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="valid",
cnn_layers=4, dilation=2, cnn_dropout_rate=0.4, cnn_do_bn_order=False,
cnn_activation_before_bn_do=False),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=50)
plot_comparison(model_names=\
'Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x4,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) BN relu DO(0.2))x4,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.4) relu BN)x4,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) BN relu DO(0.4))x4,d=2 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
min_loss=80, max_loss=120, max_epoch=50)
Observations:¶
- Again, for dilated CNN layers the order of batch-norm, ReLU and dropout does not matter
Finding optimal Dropout rate for CNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.1,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.2,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=200)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.3,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=200)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.4,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=300)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.5,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=300)
plot_comparison(model_names =
['Spec CNN(200 (3,1) DO(0.1) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.3) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.4) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.5) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=120, min_loss=90)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=False),
rnn_type=M.RNNType.GRU, rnn_layers=1), epochs=150)
plot_comparison(model_names =
['Spec CNN(200 (3,1) DO(0.2) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.25) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (3,1) DO(0.3) relu BN)x12 BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)'],
max_loss=120, min_loss=90)
Observations:¶
- Dropout rate of 0.25 appears to be optimal for CNN layers
Testing the importance and order of Batch Normalization, ReLU and Dropouts in the RNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = False, rnn_dropout_rate = 0), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = True, rnn_dropout_rate = 0), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = False, rnn_dropout_rate = 0.2), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = False, rnn_dropout_rate = 0.5), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = True, rnn_dropout_rate = 0.2), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = True, rnn_dropout_rate = 0.1), epochs=30)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4,
rnn_bn = True, rnn_dropout_rate = 0.3), epochs=30)
plot_comparison(model_names=
['Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 relu(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 relu DO(0.5)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.1)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.3)(:-1)) relu BN DO(0.2) TD(D)'],
max_loss=120, min_loss=100, max_epoch=30)
Observations:¶
- Dropout is much more essential than Batch Normalization as omitting Dropout while keeping Batch Normalization (yellow line above) degrades performance much more than omitting Batch Normalization while keeping Dropout (green and red lines above)
- In fact, with dropout rate of 0.2, omitting Batch Normalization barely affects performance (green line vs. dark purple line above)
- The dropout rate of 0.2 appears to be optimal for RNN layers regardless of the presense of Batch Normalization (green vs. red line above without BatchNorm, last three lines with BatchNorm)
- According to the original paper, Batch Normalization may be expected to make Dropout unneccesary or at least, allow to lower dropout rate. For the RNN layers of this task the opposite is observed: in the presense of Dropout Batch Normalization makes little difference either with regard to model performance or with regard to the optimal dropout rate.
Determining optimal number of RNN layers¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=1, rnn_dropout_rate = 0.2), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=2, rnn_dropout_rate = 0.2), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=4, rnn_dropout_rate = 0.2), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=6, rnn_dropout_rate = 0.2), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=8, rnn_dropout_rate = 0.2), epochs=200)
train_model(M.RNNModel(cnn_config=M.CNNConfig(), rnn_layers=12, rnn_dropout_rate = 0.2), epochs=200)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x2 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x6 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x12 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)'],
min_loss=95, max_loss=130)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x1) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x2 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x6 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x12 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)'],
min_loss=95, max_loss=110)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x6 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)'],
min_loss=95, max_loss=110)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.2) BN) BD(concat) CuDNNGRU(200 x12 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)'],
min_loss=95, max_loss=110)
Observations:¶
- While RNN networks are expected to be shallow, optimal number of RNN layers (with one CNN layer) appears to be around 8 for this task
Testing the importance and order of Batch Normalization, ReLU and Dropouts between RNN and Time-Distributed Dense¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = False, do_bn_order = False), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = False, do_bn_order = True), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = True, do_bn_order = False), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = True, do_bn_order = True), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = True, bn = False), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
activation_before_bn_do = False, dropout_rate = 0), epochs=50)
plot_comparison(model_names=\
[ 'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) DO(0.2) relu BN TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu BN DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu DO(0.2) BN TD(D)'],
min_loss = 95, max_loss=110)
Observations:¶
- Here at last, the canonical BatchNorm-ReLU-Dropout order (blue line on the chart above) performs almost as well as less canonical order
plot_comparison(model_names=\
[ 'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) relu DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu TD(D)'],
min_loss = 95, max_loss=110)
Observations:¶
- Here again, omitting BatchNormalization (yellow line) degrades performance a bit less than omitting Dropout (green line)
Determining optimal Dropout rate before Time-Distributed-Dense¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.1), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.2), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.4), epochs=50)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.5), epochs=50)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.1) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.4) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.5) TD(D)'],
min_loss = 90, max_loss=115)
train_model(M.RNNModel(cnn_config=M.CNNConfig(cnn_dropout_rate=0.25), rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.25), epochs=50)
plot_comparison(model_names=\
['Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.2) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.25) TD(D)',
'Spec CNN(200 (11,2) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)'],
min_loss = 90, max_loss=115)
Observations:¶
- Dropout rate of 0.3 appears optimal before Time-Distributed-Dense
Combining CNN and RNN¶
Previous experiments have shown that:
- with one RNN layer, optimal number of CNN layers is 12 with dropout rate 0.25, dilutions or DenseNet in CNN do not improve performance
- with one CNN layer, optimal number of RNN layers is 8 with dropout rate 0.2
- optimal dropout rate before TDD is 0.3
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=1, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_layers=8, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_layers=1, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_layers=8, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=500)
plot_comparison(model_names=[
'Spec CNN(200 (3,1) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.3) TD(D)'
], max_loss=105)
Observations:¶
- It appears that deep CNN and RNN layers are interchangeable: Deep CNN + Shallow RNN displays about the same performance as Shallow CNN + Deep RNN
plot_comparison(model_names=[
'Spec CNN(200 (3,1) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)'
], max_loss=105, max_epoch=400)
Observations:¶
- ... and Deep CNN + Deep RNN gives about the same minimal validation loss as either Deep CNN + Shallow RNN or Shallow CNN + Deep RNN
Trying Dense RNN¶
- Previous experiments have shown that optimal number of CNN layers with one RNN layer is 12 and optimal number of RNN layers with one CNN layer is 8
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"), epochs=150)
plot_comparison(model_names=[
'Spec CNN(200 (3,1) relu DO(0.25) BN) BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU(200 x1) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU(200 x8 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final'
], max_loss=105, max_epoch=150)
Observations:¶
- Deep CNN + Dense RNN (in a sense of DenseNet paper) shows best performance so far
Fine-tuning Dense RNN¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=4, rnn_dropout_rate = 0.3,
dropout_rate=0.3), epochs=100)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.35,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=4, rnn_dropout_rate = 0.3,
dropout_rate=0.3), epochs=300)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=6, cnn_dropout_rate=0.35, cnn_dense=True,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=4, rnn_dropout_rate = 0.3,
dropout_rate=0.3), epochs=100)
plot_comparison(model_names=[
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.3)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.35) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.3)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN_DENSE(200 (3,1) DO(0.35) relu BN)x6 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.3)(:-1)) BN relu DO(0.3) TD(D)'
], max_loss=105)
Observations:¶
- Raising dropout rate for DenseRNN or for CNN part does not help
- Dense CNN part also shows worse performance
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=300, rnn_layers=4, rnn_dropout_rate = 0.3,
dropout_rate=0.3, name_suffix="Final"), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=300, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_layers=5, rnn_dropout_rate = 0.2,
dropout_rate=0.3), epochs=150)
plot_comparison(model_names=[
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(300 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(300 x4 BN relu DO(0.3)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(200 x5 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D)'
], max_loss=105)
Observations:¶
- It appears beneficial for model performance to raise the number of DenseRNN units to 250 (green and yellow lines), but not more
- Decreasing the depth of CNN part to 8 layers (green line) also improves performance slightly
GRU vs LSTM:¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.LSTM,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"), epochs=150)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.LSTM,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.5,
dropout_rate=0.3, name_suffix="Final"), epochs=150)
plot_comparison(model_names=[\
'Spec CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNLSTM_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'Spec CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNLSTM_DENSE(250 x4 BN relu DO(0.5)(:-1)) BN relu DO(0.3) TD(D) Final' \
],min_loss=90, max_loss=120, max_epoch=100)
Observations:¶
- A model with Dense LSTM instead of Dense GRU (all other parameters left the same) has about 20% more trainable parameters, trains faster, overfits faster, and shows a bit larger minimal validation loss
- Raising dropout rate to 0.5 in the Dense LSTM layers gives about the same minimal validation loss as with the Dense GRU model (at 0.2 dropout at GRU layers) but the minimum is reached a bit sooner. It is, indeed, somewhat unexpected that a larger model with higher dropout rate trains faster.
Spectrogram vs MFCC:¶
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"), spectrogram=False, epochs=150)
plot_comparison(model_names=[\
'Spec CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'MFCC CNN(200 (3,1) relu DO(0.25) BN)x12 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final'
],min_loss=90, max_loss=120, max_epoch=120)
train_model(M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=8, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"), spectrogram=False, epochs=150)
plot_comparison(model_names=[\
'Spec CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final',
'MFCC CNN(200 (3,1) relu DO(0.25) BN)x8 BD(concat) CuDNNGRU_DENSE(250 x4 BN relu DO(0.2)(:-1)) BN relu DO(0.3) TD(D) Final'
],min_loss=90, max_loss=120, max_epoch=100)
Observations:¶
- A model with MFCC input trains almost identically to a model with the same architecture and parameters but with Spectrogram input.
- This confirms that, for the given task and model architecture, the MFCC is an adequate and efficient feature extration technique.
- On the other hand, the fact that given Spectrogram inputs the model does no worse than with MFCC inputs indicates that the model is quite capable of learning all the insights and principles that humans put into constructing MFCC :)
Question 1: Use the plot above to analyze the performance of each of the attempted architectures. Which performs best? Provide an explanation regarding why you think some models perform better than others.
Answer:
- First, we see that adding Time-Distributed-Dense after RNN dramatically improves performance. This could be due to each of 200-250 dimensions of RNN learning its own time pattern yet the RNN not mixing together the information of presence of different patterns. Such mixing may be essential for the task and the Time-Distributed-Dense layer does that. It would be interesting to try different depth of the Dense layer.
- Next, we find that the architecture must include at least one CNN and at least one RNN layer. Probably, a lot of information about pronounced phoneme is quite local in time, within few timesteps of the input. So a CNN is more efficient in detecting local patterns than RNN. On the other hand, some patterns must be spread along great many timesteps so only RNN can learn such patterns
- Bidirectional RNN looks both to the past and to the future in time when detecting patterns which makes its job easier compared to unidirectional RNN that only looks to the past. Let's say a sinusoidal harmonics begins. A unidirectional one would be able to detect it only later in time when it has already seen a few periods. A bidirectional RNN would see a few periods in future data and could, therefore, detect the harmonics right from the start.
- Multiple CNN layers with small kernel size perform better than small number of CNN layers with larger kernels. Stacked small kernels have the same effective span as few layers with larger kernels, yet the former are more flexible than the latter due to more linear transformations and non-linear activations.
- Dilations help construct CNNs with extremely large spans (up to 512 timesteps in the WaveNet paper). When applied to raw audio data (as in WaveNet paper) dilated CNNs can emulate Fourier transforms (in some sense, learning that sinuses and cosinuses form a good basis for audio data) or they can learn a basis system of functions that is even better suited for the task. Applying dilations to spectrogram data does not look like a natural idea and has not shown any improvement in model performance. Nearly identical performance of normal and "inverse" dilations on spectrogram data supports the understanding that using dilations on spectrograms is not a good idea. (However, the repository referenced in the project text contains implementation of dilations on MFCCs)
- Naive Dropout in the RNN part works and helps a lots. Lesser importance of BatchNorm relative to (naive) Dropout (especially near-irrelevance of BatchNorm in RNN layers) is contrary to relults presented in the Batch Normalization paper. However, Dropout prevents neurons is the same layer from relying on each other and it is not at all clear or articulated in the BatchNorm paper how BatchNorm would achieve the same. It this task, even though the inputs are pretty regular, gradient explosions, against which the BatchNorm is also a tool are, nevetheless, quite possible as the text of this notebook tells us. Perhaps the gradient clipping in the optimizer reduces the effect of BatchNorm?
- If restricted to only one CNN layer, the network requires up to a dozen RNN layers to be optimal. This suggests that quite complicated non-linear patterns have to be learned by the network.The fact that Deep CNN + Shallow RNN performs about the same as Shallow CNN + Deep RNN indicates that most of these non-linear patterns are quite local, within up to 15 (3 in kernel + 1 stride * 12 layers) timesteps
- DenseNet architecture (where output of each layer is concatenated in the input of each following layer) does not help in the CNN part (even though it was introduced for CNN in the DenseNet paper) Could this be because DenseNet interlayer connections reduce effective span of the stack of small kernel CNN layers below the most efficient size?
- In the RNN part, however, DenseNet architecture actually helps both to reduce the number of RNN layers and lower the validation loss. Unlike those in CNN, DenseNet connections cannot reduce effective span in the time dimension in RNN. Deep RNNs may be difficult to train as it is harder for gradients to flow through all those gates of RNN cells. That could be the reason DenseNet shortcut concatenations help in the RNN part of the model.
- Model with LSTM cells trains a bit faster than the same model with GRU cells. This is despite an LSTM cell having more parameters and a higher (naive) Dropout rate being needed for LSTM model to show the same minimal validation loss as its GRU sister. It is, indeed, somewhat unexpected that a larger model with higher dropout rate trains faster.
- MFCC and Spectrogram inputs produce nearly idendical performance. This shows not only that MFCC is a good feature extraction for the given task, losing no useful information compared to Spectrogram and but also that, if given a Spectrogram input, a model can learn some equivalenly good feature extraction </ul>
(IMPLEMENTATION) Final Model¶
Now that you've tried out many sample models, use what you've learned to draft your own architecture! While your final acoustic model should not be identical to any of the architectures explored above, you are welcome to merely combine the explored layers above into a deeper architecture. It is NOT necessary to include new layer types that were not explored in the notebook.
However, if you would like some ideas for even more layer types, check out these ideas for some additional, optional extensions to your model:
- If you notice your model is overfitting to the training dataset, consider adding dropout! To add dropout to recurrent layers, pay special attention to the
dropout_Wanddropout_Uarguments. This paper may also provide some interesting theoretical background. - If you choose to include a convolutional layer in your model, you may get better results by working with dilated convolutions. If you choose to use dilated convolutions, make sure that you are able to accurately calculate the length of the acoustic model's output in the
model.output_lengthlambda function. You can read more about dilated convolutions in Google's WaveNet paper. For an example of a speech-to-text system that makes use of dilated convolutions, check out this GitHub repository. You can work with dilated convolutions in Keras by paying special attention to thepaddingargument when you specify a convolutional layer. - If your model makes use of convolutional layers, why not also experiment with adding max pooling? Check out this paper for example architecture that makes use of max pooling in an acoustic model.
- So far, you have experimented with a single bidirectional RNN layer. Consider stacking the bidirectional layers, to produce a deep bidirectional RNN!
All models that you specify in this repository should have output_length defined as an attribute. This attribute is a lambda function that maps the (temporal) length of the input acoustic features to the (temporal) length of the output softmax layer. This function is used in the computation of CTC loss; to see this, look at the add_ctc_loss function in train_utils.py. To see where the output_length attribute is defined for the models in the code, take a look at the sample_models.py file. You will notice this line of code within most models:
model.output_length = lambda x: xThe acoustic model that incorporates a convolutional layer (cnn_rnn_model) has a line that is a bit different:
model.output_length = lambda x: cnn_output_length(
x, kernel_size, conv_border_mode, conv_stride)In the case of models that use purely recurrent layers, the lambda function is the identity function, as the recurrent layers do not modify the (temporal) length of their input tensors. However, convolutional layers are more complicated and require a specialized function (cnn_output_length in sample_models.py) to determine the temporal length of their output.
You will have to add the output_length attribute to your final model before running the code cell below. Feel free to use the cnn_output_length function, if it suits your model.
Loading the final model in MFCC and Spectrogram versions¶
model_spec = T.load_model(get_gen(spectrogram=True),
M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"))
model_mfcc = T.load_model(get_gen(spectrogram=False),
M.RNNModel(cnn_config=M.CNNConfig(kernel_size=3, conv_stride=1, conv_border_mode="same",
cnn_layers=12, cnn_dropout_rate=0.25,
cnn_activation_before_bn_do=True,
cnn_do_bn_order=True),
bd_merge=M.BidirectionalMerge.concat,
rnn_type=M.RNNType.GRU,
rnn_dense=True, rnn_units=250, rnn_layers=4, rnn_dropout_rate = 0.2,
dropout_rate=0.3, name_suffix="Final"))
Question 2: Describe your final model architecture and your reasoning at each step.
Answer:
- The model has the CNN part followed by the RNN part followed by Time-Distributed Dense
- The CNN part has 12 layers of 200 filters each with kernel size 3 and stride 1. Linear kernels are followed by ReLU activations, Dropout with rate 0.25 and Batch Normalization layers (yes, in such non-canonical order).
- The RNN part is based on GRU cells and begins with Bidirectional layer with concatenation merge. It is followed by three more GRU layers with 250 units each. RNN layer is followed by canonical sequence of Batch Normalization, ReLU activation and (naive) Dropout with rate of 0.2. Output of each RNN layer is concatenated in the input of each following RNN layers (DenseNet architecture).
- The Time-Distributed Dense part is preceeded, again, by canonical sequence of Batch Normalization, ReLU activation and Dropout with rate of 0.3.
- Softmax activation concludes the model.
- The model is trained with CTC loss.
The reasoning for each step is provided in the Observations sections above as well as in the answer to Question 1.
STEP 3: Obtain Predictions¶
We have written a function for you to decode the predictions of your acoustic model. To use the function, please execute the code cell below.
from IPython.display import Audio
def get_predictions(spectrogram, model, partition, index, **kwargs):
audio_path = T.get_predictions(get_gen(spectrogram=spectrogram, shuffle=False),
model,
partition=partition, index=index, **kwargs)
return audio_path
Audio(get_predictions(True, model_spec, 'train', 0))
Audio(get_predictions(True, model_spec, 'validation', 0))
Use the code cell below to obtain the transcription predicted by your final model for the first example in the training dataset.
Audio(get_predictions(False, model_mfcc, 'train', 0))
Use the next code cell to visualize the model's prediction for the first example in the validation dataset.
Audio(get_predictions(False, model_mfcc, 'validation', 0))
def print_predictions(number):
print()
print("*"*25, " MFCC & SPEC - Training ", "*"*25, '\n')
for i in range(number):
get_predictions(True, model_spec, 'train', i, print_line=False)
get_predictions(False, model_mfcc, 'train', i, omit_true=True)
print()
print()
print("*"*25, " MFCC & SPEC - Validation ", "*"*25, '\n')
for i in range(number):
get_predictions(True, model_spec, 'validation', i, print_line=False)
get_predictions(False, model_mfcc, 'validation', i, omit_true=True)
print()
print_predictions(20)
One standard way to improve the results of the decoder is to incorporate a language model. We won't pursue this in the notebook, but you are welcome to do so as an optional extension.
If you are interested in creating models that provide improved transcriptions, you are encouraged to download more data and train bigger, deeper models. But beware - the model will likely take a long while to train. For instance, training this state-of-the-art model would take 3-6 weeks on a single GPU!